Image generating apparatus, image generating method and computer readable recording medium for recording program for generating new image from images related to reference image

ABSTRACT

In an image generating apparatus according to an embodiment of the present invention, the reference image selecting unit selects an image as a reference image. The image extracting unit extracts at least one related image related to the reference image. The image generating unit generates a new composite image by using the reference image and the at least one related image according to a layout including a definition of at least one kind of information selected from a group including a background image, the number of images to be used, a size of each image to be used, and positional relation among images to be used. The basic control unit issues an instruction to make the reference image selecting unit select an image being displayed as the reference image and make the image generating unit generate the new composite image based on the reference image.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Division of application Ser. No. 14/827,434 filedAug. 17, 2015, the entire contents of which are incorporated herein byreference.

This application is based upon and claims the benefit of priority under35 USC 119 of Japanese Patent Application No. 2014-166818 filed on Aug.19, 2014, the entire disclosure of which is incorporated herein byreference in its entirety for all purposes.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to an image generating apparatus, an imagegenerating method and a computer readable recording medium recording aprogram thereon.

2. Description of the Related Art

Conventionally, a technique called a collage has been known as a methodfor making use of images taken by a digital camera.

The collage is a method for generating a new image by designingarrangements such as sizes, positions or angles of a plurality of imagesand attaching them on a background image, thereby improving visualeffect. In general, a user can make and enjoy a collage image or post iton the web to make the public enjoy it.

As an example of technologies relating to the collage, Japanese PatentApplication Laid-Open Publication No. 2006-80632 published on Mar. 23,2006 discloses a technology for relatively easily generating a compositeimage by laying out a plurality of images selected by a user.

SUMMARY OF THE INVENTION

According to an embodiment of the present invention, an image generatingapparatus is provided. The image generating apparatus includes an imageselecting section configured to select an image as a reference image, anextracting section configured to extract at least one related imagewhich is related to the reference image selected by the image selectingsection, a generating section configured to generate a new compositeimage by using the reference image selected by the image selectingsection and the at least one related image extracted by the extractingsection according to a layout, the layout comprising a definition of atleast one kind of information selected from a group comprising abackground image, the number of images to be used to generate acomposite image, a size of each image to be used to generate a compositeimage, and positional relation among images to be used to generate acomposite image, and an instructing section configured to issue aninstruction to make the image selecting section select an image which isbeing displayed on a display screen as the reference image and make thegenerating section generate the new composite image based on theselected reference image.

According to an embodiment of the present invention, an image generatingmethod performed by an image generating apparatus is provided. The imagegenerating method includes selecting an image as a reference image,extracting at least one related image which is related to the selectedreference image, generating a new composite image by using the selectedreference image and the extracted at least one related image accordingto a layout, the layout comprising at least one definition ofinformation selected from a group comprising a background image, thenumber of images to be used to generate a composite image, a size ofeach image to be used to generate a composite image, and positionalrelation among images to be used to generate a composite image, andissuing an instruction to select an image which is being displayed on adisplay screen as the reference image and generate the new compositeimage based on the selected reference image.

According to an embodiment of the present invention, a non-transitorycomputer-readable recording medium for recording a program readable by acomputer is provided. The program controls the computer to executefunctions including an image selecting function of selecting an image asa reference image, an extracting function of extracting at least onerelated image which is related to the reference image selected by theimage selecting function, a generating function of generating a newcomposite image by using the reference image selected by the imageselecting function and the at least one related image extracted by theextracting function according to a layout, the layout comprising atleast one definition of information selected from a group comprising abackground image, the number of images to be used to generate acomposite image, a size of each image to be used to generate a compositeimage, and positional relation among images to be used to generate acomposite image, and an instructing function of issuing an instructionto make the image selecting function select an image which is beingdisplayed on a display screen as the reference image and making theimage generating function generate the new composite image based on theselected reference image.

The above and further objects and novel features of the presentinvention will more fully appear from the following detailed descriptionwhen the same is read in conjunction with the accompanying drawings. Itis to be expressly understood, however, that the drawings are for thepurpose of illustration only and are not intended as a definition of thelimits of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will more sufficiently be understood by thefollowing detailed description and the accompanying drawings, which are,however, exclusively for explanation and do not limit the scope of thepresent invention.

Here:

FIGS. 1A and 1B are schematic diagrams showing exterior features of animage generating apparatus according to an embodiment of the presentinvention. FIG. 1A shows a state in which a rear surface unit 1B is in astandard position and FIG. 1B shows a state in which the rear surfaceunit 1B is rotated about a hinge.

FIG. 2 is a block diagram showing a hardware configuration of an imagegenerating apparatus according to an embodiment of the presentinvention.

FIG. 3 is a functional block diagram showing a functional configurationof the image generating apparatus for executing an image generationprocess.

FIG. 4 is a schematic diagram showing exemplary layouts.

FIG. 5 is a schematic diagram showing an example of a process forextracting an image group executed by an image extracting unit 52.

FIG. 6 is a schematic diagram showing an example of a process forextracting an image group executed by the image extracting unit 52.

FIG. 7 is a flow chart for explaining the image generation processexecuted by the image generating apparatus shown in FIGS. 1A and 1Bincluding the functional configuration shown in FIG. 3.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

Hereinafter, embodiments of the present invention will be described withreference to the accompanying drawings.

FIGS. 1A and 1B are schematic diagrams showing exterior features of animage generating apparatus 1 according to an embodiment of the presentinvention. FIG. 1A shows a state in which a rear surface unit 1B is in astandard position and FIG. 1B shows a state in which the rear surfaceunit 1B is rotated about a hinge.

FIG. 2 is a block diagram showing a hardware configuration of the imagegenerating apparatus 1 according to an embodiment of the presentinvention.

For example, the image generating apparatus 1 is realized by a digitalcamera. In the present embodiment, the image generating apparatus 1includes a main body 1A and the rear surface unit 1B forming a rearsurface of the main body 1A. The rear surface unit 1B is rotatablyconnected to the main body 1A by the hinge.

The image generating apparatus 1 includes a CPU (Central ProcessingUnit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13,a bus 14, an I/O interface 15, an imaging unit 16, an input unit 17, anoutput unit 18, a storage unit 19, a communication unit 20, and a drive21.

The CPU 11 executes various processes according to programs stored inthe ROM 12 or loaded in the RAM 13 from the storage unit 19.

In the RAM 13, there are stored data necessary for the CPU 11 to executevarious processes, and the like.

The CPU 11, the ROM 12 and the RAM 13 are connected to each other viathe bus 14. The I/O interface 15 is also connected to the bus 14. Theimaging unit 16, the input unit 17, the output unit 18, the storage unit19, the communication unit 20, and the drive 21 are connected to the I/Ointerface 15.

The imaging unit 16 includes an optical lens unit and an image sensor(not shown in the drawing).

The optical lens unit includes lenses for collecting light to photographa subject such as a focus lens and a zoom lens.

The focus lens forms an image of a subject on a light-receiving surfaceof the image sensor. The zoom lens freely changes the focal lengthwithin a predetermined range.

Further, the optical lens unit is provided with a peripheral circuit toadjust parameters such as focusing, exposure, and/or white balancing, asnecessary.

The image sensor includes a photoelectric conversion element, an AFE(Analog Front End), and the like.

The photoelectric conversion element includes a CMOS (ComplementaryMetal Oxide Semiconductor) type photoelectric conversion element, forexample. A subject's image is input to the photoelectric conversionelement from the optical lens unit. The photoelectric conversion elementperforms photoelectric conversion (image capturing) of the subject'simage and accumulates image signals for a predetermined period of time.The photoelectric conversion element provides the AFE with theaccumulated image signals sequentially.

The AFE executes various signal processing operations such as A/D(Analog/Digital) conversion on the analog image signals. Digital signalsare generated by the signal processing operations and output as outputsignals of the imaging unit 16.

The output signal of the imaging unit 16 is hereinafter referred to as“a captured image”. The captured image is supplied to the CPU 11.

The input unit 17 includes a touch panel stacked on a display 18 a ofthe output unit 18 and a variety of information is input via the inputunit 17 in response to manipulation by a user. The touch panel may be acontact type or a non-contact type. Further, in addition to the touchpanel, the input unit 17 includes various buttons including a mainshutter button 17 a and a front shutter button 17 b for inputting avariety of information in response to manipulation by the user. In thepresent embodiment, the main shutter button 17 a is provided on the topsurface of the image generating apparatus 1 and the front shutter button17 b is provided on a front surface of the image generating apparatus 1(i.e. a surface from which the optical lens unit protrudes). In the casethat the image generating apparatus 1 is in a shooting mode for shootingan image, both of the main shutter button 17 a and the front shutterbutton 17 b function as a shutter button for instructing to shoot animage. In the case that the image generating apparatus 1 is in aplayback mode for playing the shot image back, the main shutter button17 a functions as a direct button for instructing to switch to theshooting mode for shooting an image and the front shutter button 17 bfunctions as a direct button for instructing to execute a process foreasily generating a collage image (a direct collage mode which will bedescribed later).

The output unit 18 includes a display 18 a, a speaker, or the like, andoutputs images or voices. In the present embodiment, a body rear surfaceof the image generating apparatus 1 is formed by the rear surface unit1B which is different from the main body 1A. The rear surface unit 1Bincludes the display 18 a and is connected to the main body 1A by thehinge with a rotation axis along an upper edge of the body rear surface.Thus, the display 18 a stands alongside the body rear surface in a stateof standard usage for taking a snapshot or a landscape picture (see FIG.1A) and the rear surface unit 1B is rotated about the hinge and thedisplay 18 a stands alongside a body front surface in a state of specialusage for taking a self-portrait, for example (see FIG. 1B).

The storage unit 19 includes a hard disk, a DRAM (Dynamic Random AccessMemory), or the like and various image data is stored therein.

The communication unit 20 controls communication with different devices(not shown in the drawing) via a network such as Internet.

A removable media 31 including a magnetic disk, an optical disk, amagneto-optical disc, a semiconductor memory, or the like, is mounted onthe drive 21. A program read out from the removable media 31 by thedrive 21 is installed in the storage unit 19 as necessary. Similarly tothe storage unit 19, the removable media 31 stores various data such asthe image data stored in the storage unit 19.

FIG. 3 is a functional block diagram showing a functional configurationof the image generating apparatus 1 for executing an image generationprocess.

The image generation process means a series of actions taken in order toselect a reference image and the other images automatically or manuallyfrom a plurality of images and generate one image (a collage image) bylaying out the selected images.

In the case of executing the image generation process, a reference imageselecting unit 51, an image extracting unit 52, a layout selecting unit53, an image generating unit 54, a basic control unit 55, and a displaycontrol unit 56 of the CPU 11 function.

In an area of the storage unit 19, an image storage unit 71 and a layoutstorage unit 72 are configured.

In the image storage unit 71, there is stored data of images which willbe used for generation of a collage image in the image generationprocess. The data stored in the image storage unit 71 can be data ofimages shot by the image generating apparatus 1 or data input from adifferent device, for example.

In the layout storage unit 72, there is stored data of layouts whichwill be used for generation of a collage image in the image generationprocess.

FIG. 4 is a schematic diagram showing exemplary layouts. In FIG. 4, fourlayouts stored in the layout storage unit 72 (layouts A to D) are shownas an example.

As shown in FIG. 4, a layout is a template including layout frames intowhich a predetermined number of images including a reference image areinserted and a background image. For each layout, the number of layoutframes (i.e. the number of images forming a collage image) is set. Foreach of the layout frames, positional relation (central positions),priorities, sizes and rotation angles of images to be inserted are set.In the present embodiment, a “main layout frame” having the highestpriority is a layout frame having the biggest size. A priority of alayout frame different from the main layout frame is higher if it ispositioned in the left side and in the upper side. For generation of acollage image, a reference image is disposed in the main layout frameand the other images are disposed in descending order of temporalrelationship to the reference image (which will be described later) inthe layout frames in descending order of priority. The data of layoutsstored in the layout storage unit 72 are prepared beforehand.

The reference image selecting unit 51 selects an image which will be areference for generation of a collage image (the reference image). Inthe present embodiment, two kinds of modes for generation of a collageimage are available, i.e. a first quick collage mode in which areference image to be used for the collage image is automaticallyselected and a second quick collage mode in which the reference image tobe used for the collage image is manually selected. In the case that thefirst quick collage mode is selected, the reference image selecting unit51 automatically selects an image which is being displayed by thedisplay 18 a as the reference image. In the case that the second quickcollage mode is selected, the reference image selecting unit 51designates an image which the user manually selects as the referenceimage. Further, in the present embodiment, a direct collage mode whichsimplifies the user's manipulation in the first quick collage mode isalso available. In the case that the front shutter button 17 b ismanipulated in the playback mode, the image generating apparatus 1 isswitched to the direct collage mode. In the direct collage mode,selection of a layout by the layout selecting unit 53 is automaticallyperformed in addition to the selection of the reference image by thereference image selecting unit 51. In the direct collage mode accordingto the present embodiment, the reference image selecting unit 51automatically selects an image which is being displayed by the display18 a as the reference image and the layout selecting unit 53automatically selects a layout used for a collage image generated lasttime.

The image extracting unit 52 acquires time-related information of eachimage. More specifically, the image extracting unit 52 acquires theshooting date (year-month-date) and time of the reference image inseconds.

Further, the image extracting unit 52 extracts an image group to be usedfor generation of the collage image with the reference image based onthe temporal relationship to the reference image. More specifically,according to the number of layout frames of the selected layout, theimage extracting unit 52 acquires a group of images temporally adjacentwith the reference image as the center (for example, in the case thatthe number of layout frames is four, total seven images consisting ofthe reference image, three images older than the reference image andthree images newer than the reference image) from the image storage unit71. In other words, the image extracting unit 52 acquires a series ofimages with the reference image as the center by acquiring an imagegroup consisting of the set number of images (for example, four images)of which shooting time is close to that of the reference image in whichthe reference image is the newest image, an image group consisting ofthe set number of images of which shooting time is close to that of thereference image in which the reference image is the second to thenewest, an image group consisting of the set number of images of whichshooting time is close to that of the reference image in which thereference image is the third to the newest, and an image groupconsisting of the set number of images of which shooting time is closeto that of the reference image in which the reference image is theoldest image.

Further, the image extracting unit 52 calculates interval time andseparation time from the acquired shooting date and time of the seriesof images. The interval time is a difference in shooting time betweentwo adjacent images and the separation time is a difference in shootingtime between the reference image and each of other image(s). The imageextracting unit 52 calculates a total interval time (the sum of intervaltime) and a total separation time (the sum of separation time) for eachof the image groups.

The image extracting unit 52 calculates the total interval time and thetotal separation time for each of the image groups included in theseries of images and extracts an image group having the closest temporalrelationship to the reference image from the image groups based on thetotal interval time and the total separation time. In the presentembodiment, the image extracting unit 52 extracts an image group havingthe lowest total separation time as an image group having the closesttemporal relationship to the reference image. In the case that at leasttwo image groups have the lowest total separation time, the imageextracting unit 52 extracts an image group having the lowest totalinterval time from all of the image groups as an image group having theclosest temporal relationship to the reference image. In the case thatat least two image groups have the lowest total interval time, the imageextracting unit 52 extracts an image group including the oldest imagefrom all of the image groups as an image group having the closesttemporal relationship to the reference image. An image group selected asabove is used for generation of the collage image.

FIGS. 5 and 6 are schematic diagrams showing exemplary processes forextracting an image group executed by an image extracting unit 52. InFIGS. 5 and 6, each of N and M is a natural number indicating a numberof each image.

As shown in FIGS. 5 and 6, in the case that a collage image is generatedby using four images including a reference image, total seven imagesconsisting of the reference image, three images older than the referenceimage and three images newer than the reference image are read from theimage storage unit 71 into the RAM 13 by referring to shooting date andtime of each image (Procedure 1). For the seven images, first to fourthselection patterns can be considered as image groups each of whichincludes four adjacent images.

Then, for the seven images read at Procedure 1, interval time iscalculated (Procedure 2).

Then, separation time from the shooting date and time of the referenceimage is calculated from the interval time calculated at Procedure 2(Procedure 3).

Then, from the interval time calculated at Procedure 2 and theseparation time calculated at Procedure 3, a total interval time and atotal separation time are calculated for each of the selection patterns(Procedure 4).

After that, the total separation time calculated at Procedure 4 issorted in ascending order of value and a selection pattern having thelowest total separation time is extracted as an image group having theclosest temporal relationship to the reference image (Procedure 5).

In the case that seven images are read based on a reference image ofwhich shooting date and time is different and at least two selectionpatterns have the lowest total separation time at Procedure 5, the totalinterval time calculated at Procedure 4 is sorted in ascending order ofvalue and a selection pattern having the lowest total interval time isextracted as an image group having the closest temporal relationship tothe reference image (Procedure 6).

By the above procedures, it is possible to easily extract images whichthe user wants to use from a plurality of images.

Returning to FIG. 3, the layout selecting unit 53 selects a layout whichthe user manually selects as a layout to be used for generation of acollage image in the first quick collage mode and the second quickcollage mode. Further, in the direct collage mode, the layout selectingunit 53 automatically selects a layout which was previously selected forgeneration of a collage image. For example, the layout selecting unit 53automatically selects the layout selected for the collage imagegenerated last time as a layout to be used for generation of a newcollage image. Alternatively, in the direct collage mode, the user maydesignate a default layout which is automatically selected.

The image generating unit 54 generates a collage image by disposing thereference image selected by the reference image selecting unit 51 andthe images of the image group extracted by the image extracting unit 52in the layout frames of the layout selected by the layout selecting unit53. The image generating unit 54 disposes the reference in the mainlayout frame and the other images in descending order of temporalrelationship to the reference image (i.e. in descending order ofproximity of shooting time) in the remaining layout frames in descendingorder of priority. Further, the image generating unit 54 changes thesize, the central position and the rotation angle of each imageaccording to a layout frame corresponding to the image and then disposesit in the layout frame. Thus, one collage image is generated from thereference image, the images of the image group and the layout.

The basic control unit 55 executes basic control processes such as aprocess of generating a menu screen, processes corresponding to variousoperations selected from the menu screen, and processes corresponding tomanipulation of various buttons included in the input unit 17.

The display control unit 56 executes control the display 18 a to displayvarious images and the menu screen for the image generation process. Forexample, the display control unit 56 controls the display 18 a todisplay reference image candidates sequentially or a list of layoutcandidates. Further, the display control unit 56 controls the display 18a to display the collage image generated by the image generating unit54.

FIG. 7 is a flow chart for explaining the image generation processexecuted by the image generating apparatus 1 shown in FIGS. 1A and 1Bincluding the functional configuration shown in FIG. 3.

Upon switching the image generating apparatus 1 to the playback mode bymanipulation of the input unit 17 and displaying images stored in theimage storage unit 71, the image generation process is started.

At Step S1, the basic control unit 55 determines whether or not a userperforms an operation of selecting one of images displayed on a displayscreen.

In the case that the user performs the operation of selecting one of theimages displayed on the display screen, the determination at Step S1 is“YES” and the process proceeds to Step S2.

On the other hand, in the case that the user does not perform theoperation of selecting one of the images displayed on the displayscreen, the determination at Step S1 is “NO” and the process proceeds toStep S3.

At Step S2, the basic control unit 55 displays the selected image on thedisplay screen.

At Step S3, the basic control unit 55 determines whether or not thefront shutter button 17 b is manipulated.

In the case that the front shutter button 17 b is not manipulated, thedetermination at Step S3 is “NO” and the process proceeds to Step S4.

On the other hand, in the case that the front shutter button 17 b ismanipulated, the determination at Step S3 is “YES” and the processproceeds to Step S12.

At Step S4, the basic control unit 55 determines whether or not a menumanipulation is performed.

In the case that no menu manipulation is performed, the determination atStep S4 is “NO” and the process proceeds to Step S5.

On the other hand, in the case that the menu manipulation is performed,the determination at Step S4 is “YES” and the process proceeds to StepS6.

At Step S5, the basic control unit 55 executes operations other than themenu manipulation such as switch to the shooting mode.

After Step S5, the image generation process ends.

At Step S6, the basic control unit 55 determines whether or not the userselects the first quick collage mode.

In the case that the first quick collage mode is not selected by theuser, the process proceeds to Step S7.

In the case that the first quick collage mode is selected by the user,the process proceeds to Step S10.

At Step S7, the basic control unit 55 determines whether or not the userselects the second quick collage mode.

In the case that the second quick collage mode is not selected by theuser, the process proceeds to Step S8.

In the case that the second quick collage mode is selected by the user,the process proceeds to Step S9.

At Step S8, the basic control unit 55 executes an operation according toa function selected by the menu manipulation (such as copy of an imageor a slide show).

After Step S8, the image generation process ends.

At Step S9, the reference image selecting unit 51 receives the user'sselection of a reference image. Here, an image selected by the user isdesignated as the reference image by the reference image selecting unit51.

After Step S9, the process proceeds to Step S11.

At Step S10, the reference image selecting unit 51 selects an imagewhich is being displayed by the display 18 a as the reference image.

At Step S11, the layout selecting unit 53 receives the user's selectionof a layout from a selecting operation which the user performs in aselection screen in which a list of thumbnails showing layouts isdisplayed. Here, the layout selected by the user is designated as alayout to be adopted for a collage image by the layout selecting unit53. The display type of the selection screen is not limited to the listof thumbnails and may be a list of names indicating the layouts.

At Step S12, the reference image selecting unit 51 selects an imagewhich is being displayed by the display 18 a as the reference image.

At Step S13, the layout selecting unit 53 selects the layout selectedfor the collage image generated last time as a layout to be used forgeneration of a collage image.

At Step S14, the image extracting unit 52 extracts one or more images tobe used for generation of the collage image. More specifically, theimage extracting unit 52 acquires time-related information of eachimage. Further, based on the time-related information of each image, theimage extracting unit 52 acquires a series of images temporally adjacentwith the reference image as the center from the image storage unit 71.Further, the image extracting unit 52 calculates interval time andseparation time from the shooting date and time of the acquired seriesof images. The image extracting unit 52 also calculates a total intervaltime and a total separation time for each of image groups included inthe series of images. The image extracting unit 52 extracts an imagegroup having the closest temporal relationship to the reference imagefrom the image groups based on the total interval time and the totalseparation time.

At Step S15, the image generating unit 54 generates a collage image bydisposing the reference image selected by the reference image selectingunit 51 and the remaining image(s) of the image group extracted by theimage extracting unit 52 in layout frames of the layout selected by thelayout selecting unit 53.

At Step S16, the image generating unit 54 allows the generated collageimage to be stored in the image storage unit 71.

After Step S16, the image generation process ends.

According to the process described above, in the case that the user setsthe image generating apparatus 1 to the first quick collage mode, animage which is being displayed by the display 18 a is selected as thereference image and an image group having close temporal relationship tothe reference image is automatically extracted. Further, a collage imageis generated by disposing the reference image and the remaining image(s)of the image group in a layout selected by the user.

Therefore, it is possible to easily generate an effective image from aplurality of images.

Further, in the direct collage mode, in the case that the usermanipulates the front shutter button 17 b when an image is beingdisplayed by the display 18 a, the image which is being displayed by thedisplay 18 a is selected as the reference image and an image grouphaving close temporal relationship to the reference image isautomatically extracted without the user's menu manipulation. Inaddition, the layout used for the collage image generated last time isautomatically selected and a new collage image is generated by disposingthe reference image and the remaining image(s) of the image group in thelayout.

Thus, it is possible to reduce the user's effort to select a group ofimages and/or a layout and generate an effective image from a pluralityof images more easily.

(Modification 1)

In the above embodiments, an image group to be used for generation of acollage image with a reference image is extracted based on the temporalrelationship to the reference image. Alternatively, it is possible todetermine relationship between each image and the reference image basedon a factor other than time.

More specifically, relationship between an image and the reference imagecan be determined based on subjects of the images.

For example, it is possible to extract an image group to be used forgeneration of a collage image from images including a subject the sameas the face of a subject included in the reference image.

In this case, the image extracting unit 52 detects the face of thesubject included in the reference image and acquires images including aface the same as the face of the subject included in the reference imagefrom the image storage unit 71. Further, the image extracting unit 52extracts an image group having close temporal relationship to thereference image from the acquired images.

Thus, it is possible to simply generate a collage image consisting of areference image and other image(s) including the same subject as that ofthe reference image.

Further, it is also possible to determine relationship to the referenceimage based on a shooting location of each image as a factor other thantime.

For example, it is possible to extract an image group to be used forgeneration of a collage image from images shot at locations relativelyclose to a location at which the reference image is shot.

In this case, information on a shooting location may be added to eachimage by using a positioning technology such as the GPS (GlobalPositioning System). The image extracting unit 52 acquires the shootinglocation of each image and determines that an image shot at a locationcloser to the location at which the reference image is shot has closerrelationship to the reference image.

Thus, it is possible to simply generate a collage image consisting of areference image and images shot at locations close to a location atwhich the reference image is shot.

(Modification 2)

According to another modification of the above embodiments, one or moreimages shot within a predetermined period of time (for example, withinone hour) before or after shooting time of a reference image areextracted and a layout is automatically selected according to the numberof extracted image(s). A layout similar to that used for the collageimage generated last time can be automatically selected. Morespecifically, layouts can be classified into groups by kinds ofbackground and/or positional relation of layout frames and a layout canbe automatically selected from a group including the layout used for thecollage image generated last time.

Thus, it is possible to reduce the user's effort to select a group ofimages and/or a layout and generate a collage image more simply.

(Modification 3)

In the above embodiments, an image group is extracted to generate acollage image with a reference image based on the temporal relationshipto the reference image. According to another modification of the aboveembodiments, the temporal relationship is determined for every year orevery month. In other words, after masking information on year or month,the temporal relationship is determined on a monthly or shorter basis ora daily or shorter basis. Thus, images in the same period of time ofevery year or every month are determined to have close temporalrelationship.

Thus, it is possible to simply generate a collage image consisting ofimages of an event shot at the same time of every year or every month(for example, a birthday party or a monthly presentation).

(Modification 4)

According to another modification of the above embodiments, a layout isautomatically selected according to the number of images to be used fora collage image. In other words, the layout selecting unit 53 mayautomatically select one or more layouts having layout frames of thenumber corresponding to the number of images included in the image groupextracted by the image extracting unit 52.

Thus, it is possible to reduce the user's effort to select a layout andgenerate a collage image more simply.

The image generating apparatus 1 configured as described above includesthe reference image selecting unit 51, the image extracting unit 52, theimage generating unit 54, and the basic control unit 55.

The reference image selecting unit 51 selects an image as the referenceimage.

The image extracting unit 52 extracts at least one related image whichis related to the reference image selected by the reference imageselecting unit 51.

The image generating unit 54 generates a new composite image by usingthe reference image selected by the reference image selecting unit 51and the at least one related image extracted by the image extractingunit 52 according to a layout. The layout includes a definition of atleast one kind of information selected from a group including abackground image, the number of images to be used to generate acomposite image, a size of each image to be used to generate a compositeimage, and positional relation among images to be used to generate acomposite image.

The basic control unit 55 issues an instruction to make the referenceimage selecting unit 51 select an image which is being displayed on adisplay screen as the reference image and make the image generating unit54 generate the new composite image based on the selected referenceimage.

Thus, it is possible to automatically select a reference image and oneor more other images to generate a new image such as a composite image.

Therefore, it is possible to generate an effective image from aplurality of images more easily.

The image generating apparatus 1 includes the layout selecting unit 53.

In some embodiments, the layout selecting unit 53 selects a layoutaccording to a user's manipulation.

By this, it is possible to an effective image from a plurality of imagesmore easily by using a layout selected by the user.

In some embodiments, the layout selecting unit 53 selects a layoutselected for a composite image generated last time in the case that aninstruction to generate the new composite image is issued by the basiccontrol unit 55.

By this, it is possible to reduce the user's effort to select a layoutand generate an effective image from a plurality of images more easily.

In some embodiments, the image extracting unit 52 extracts at least onerelated image based on the number of images to be used to generate acomposite image. The definition of the number of images to be used togenerate a composite image is included in the layout selected by thelayout selecting unit 53.

By this, since a proper number of images can be automatically extracted,it is possible to generate an effective image from a plurality of imagesmore easily.

In some embodiments, the layout selecting unit 53 selects a layoutaccording to the number of related images extracted by the imageextracting unit 52.

By this, it is possible to automatically select a layout according tothe number of related images which are related to the reference image.

In some embodiments, the layout selecting unit 53 gives priority to alayout similar to that selected for a composite image generated lasttime when selecting a layout.

By this, it is possible to reduce the user's effort to select a layoutand generate an effective image from a plurality of images more easily.

In some embodiments, the image generating apparatus includes a playbackmode.

In the playback mode, an image to be displayed by the display may changeto another image while an image of a plurality of images is beingdisplayed by the display.

The basic control unit 55 issues an instruction to make the imagegenerating unit 54 generate a new composite image while the image changeis being performed in the playback mode.

By this, in the case that a desired image is being displayed by thedisplay 18 a, the user can instruct the image generating unit 54 togenerate a composite image. Thus, it is possible to easily generate aneffective image.

In some embodiments, while an image is being displayed in the playbackmode, generation of the new composite image is performed by the imagegenerating unit 54 in the case that an instruction to generate the newcomposite image is issued. Further, a changed image is displayed withoutthe generation of the new composite image being performed by the imagegenerating unit 54 in the case that an instruction to change the imagebeing displayed is issued.

By this, the user can easily instruct generation of a new image whilereferring to changed images.

In some embodiments, the image generating apparatus 1 includes ashooting mode.

In the shooting mode, it is possible to shoot an image.

The basic control unit 55 operates in response to manipulation of thefront shutter button 17 b functioning as a means for issuing aninstruction to generate a new composite image in the playback mode and ameans for issuing an instruction to shoot an image in the shooting mode.

By this, it is possible to allow the front shouter button 17 b, whichfunctions as a shutter button at the time of image shooting, to functionas a means for issuing an instruction to generate a new image. Thus, itis possible to improve manipulability for the user.

In some embodiments, the basic control unit 55 operates in response tomanipulation of one of a plurality of means for instructing imageshooting included in the image generating apparatus (i.e. the mainshutter button 17 a and the front shutter button 17 b).

By this, it is possible to allow one of a plurality of shutter buttonsto function as a means for instructing generation of a new image. Thus,it is possible to improve manipulability for the user.

In some embodiments, the basic control unit 55 issues an instruction toperform generation of a new composite image by manipulation of amanipulating unit installed on a predetermined location of the body ofthe image generating apparatus 1.

By this, the user can issue an instruction to generate a new image bysimple manipulation.

In some embodiments, the image extracting unit 52 extracts at least onerelated image based on time-related information included in thereference image.

By this, since at least one image having close temporal relationship tothe reference image can be extracted, it is possible to extract moresuitable images.

In some embodiments, the image extracting unit 52 extracts at least onerelated image based on location information included in the referenceimage.

By this, since at least one image having closely related in location tothe reference image can be extracted, it is possible to extract moresuitable images.

In some embodiments, the image extracting unit 52 recognizes a subjectin an image and extracts at least one related image based on therecognized subject.

By this, since at least one image of which subject is closely related tothat of the reference image can be extracted, it is possible to extractmore suitable images.

In addition, the present invention is not limited to the embodimentsdescribed above and various modifications and alternatives which canachieve the objects of the invention fall within the scope of thepresent invention.

In the embodiments described above, the layout defines spatialarrangement but the present invention is not limited to theseembodiments. According to an embodiment of the present invention, alayout defines temporal arrangement.

For example, the present invention can be applied to the case in which areference image is selected as described above with respect to theembodiments, one or more images related to the reference image areautomatically extracted, and one moving picture is generated from thereference image and the extracted image(s).

In this case, the layout frame of the layout of the above embodimentsmay correspond to an order of arrangement of the images on the time axisand the background of the layout may correspond to a pattern of effectfor connecting images.

Further, in the embodiments described above, the present invention hasbeen applied to generation of a collage image by a plurality of imagesincluding a reference image. However, the present invention is notlimited to these embodiments. The present invention can be applied tovarious things other than images. For example, the present invention maybe applied to various contents such as documents or music.

In the embodiments described above, the total separation time takespriority over the total interval time when determining the temporalrelationship between the reference image and an image group. However,the present invention is not limited to these embodiments. According tocertain embodiments, the total interval time may take priority over thetotal separation time or only one of the total separation time and thetotal interval time may be used. Further, in the embodiments describedabove, the total separation time or the total interval time is a simpleaddition of the values of the separation time or the values of theinterval time. However, according to certain embodiments, the sum of thesquare of the values of the separation time or the sum of the square ofthe values of the interval time may be used. Further, if the product ofthe values of the separation time or the product of the values of theseparation time for an image group is smaller, the image group may bedetermined to have closer relationship to the reference image.

Further, in the embodiments described above, the present invention hasbeen applied to a digital camera as an example of the image generatingapparatus 1 without limitation.

For example, the present invention can be applied to general electronicdevices having the function of image processing. More specifically, thepresent invention can be applied to a notebook type personal computer, aprinter, a television, a video camera, a portable navigation device, amobile phone, a smart phone, a portable game device, and the like.

The process sequence described above can be executed by hardware orsoftware.

In other words, the functional configuration shown in FIG. 3 is merelyan example and the present invention is not limited thereto. The imagegenerating apparatus 1 suffices if it has a function for performing theprocess sequence as a whole. Functional blocks to use to implement thisfunction are not limited to the embodiment of FIG. 3.

In addition, a functional block may be configured by a piece ofhardware, a piece of software, or their combination.

In the case that the sequence is performed by software, a programconfiguring the software is installed in a computer or the like from anetwork or a storage medium.

The computer may be a computer which is incorporated in dedicatedhardware. In addition, the computer may be a computer capable ofexecuting various functions by installing various programs therein, forexample, a general-purpose personal computer.

A storage medium for recording such a program consists of not only theremovable media 31 shown in FIG. 2 which is distributed separately fromthe apparatus's main body to provide it to users, but also a storagemedium or the like which is provided to users in a state of beingincorporated in the apparatus's main body in advance. The removablemedia 31 includes, for example, a magnetic disk (including a floppydisk), an optical disc, a magneto-optical disk, or the like. Forexample, the optical disc includes a compact disk-read only memory(CD-ROM), a digital versatile disk (DVD), a Blu-ray (RegisteredTrademark) disc, or the like. The magneto-optical disk includes aMini-Disk (MD), or the like. In addition, the storage medium which isprovided to the users in a state of being incorporated in theapparatus's main body in advance includes, for example, the ROM 12 inFIG. 2 in which a program is recorded, a hard disk included in thestorage unit 19 in FIG. 2, or the like.

Further, in the description presented here, the steps describing aprogram recorded in a recording medium include not only processes to beexecuted serially in time in order, but also processes which are notnecessarily executed serially in time but in a parallel manner orindividually.

Although some embodiments of the present invention have been describedabove, the embodiments are for illustrative purposes only and notintended to limit the technical scope of the present invention. It willbe evident that there are many other possible embodiments of the presentinvention and various modifications such as omission or substitution maybe made without departing from the spirit of the invention. Theseembodiments and modifications fall within the scope and the spirit ofthe invention described in this specification and within the scope ofthe invention as defined in the appended claims and equivalents thereof.

What is claimed is:
 1. An image generating apparatus comprising: amemory in which one or more programs are stored; and a processor,wherein the one or more computer programs cause the processor to performoperations to: select an image as a reference image from a plurality ofimages; select a most recent layout selected for a composite imagegenerated most recently as a layout for a new composite image; extractat least one related image from the plurality of images, which isrelated to the reference image selected from the plurality of images, byselecting the at least one related image from the plurality of images;and generate the new composite image different according to the selectedreference image, by using the selected reference image and the extractedat least one related image based on a layout, the layout comprising adefinition of at least one kind of information of a background image,the number of images to be used to generate the composite image, a sizeof each image to be used to generate the composite image, and positionalrelation among images to be used to generate the composite image.
 2. Theimage generating apparatus of claim 1, wherein the processor furtherperform an operation to select a layout according to the number of theextracted at least one related image.
 3. The image generating apparatusof claim 2, wherein the operation to select a layout according to thenumber of the extracted at least one related image comprises givingpriority to a layout similar to the most recent layout selected for thecomposite image generated most recently.
 4. An image generatingapparatus, comprising: an instructing unit; a display unit; a memory inwhich one or more programs are stored; and a processor, wherein,according to an instruction to generate a new composite image issued bythe instructing unit, the one or more computer programs cause theprocessor to perform operations to: select an image as a referenceimage; extract at least one related image which is related to theselected reference image; and generate the new composite image by usingthe selected reference image and the extracted at least one relatedimage based on a layout, the layout comprising a definition of at leastone kind of information of a background image, the number of images tobe used to generate the composite image, a size of each image to be usedto generate the composite image, and positional relation among images tobe used to generate the composite image, the image generating apparatushas a playback mode where an image to be displayed by the display unitis changed to another image while an image of a plurality of images isbeing displayed by the display unit, and the instructing unit issues theinstruction to generate the new composite image at arbitrary timingwhile the image change is being performed in the playback mode, andwhile an image is being displayed in the playback mode, the processorperforms the operation to generate the new composite image in the casethat the instructing unit issues the instruction to generate the newcomposite image, and a changed image is displayed without the generationof the new composite image being performed in the case that theinstructing unit issues an instruction to change the image beingdisplayed.
 5. An image generating apparatus, comprising: an instructingunit; a display unit; a memory in which one or more programs are stored;and a processor, wherein, according to an instruction to generate a newcomposite image issued by the instructing unit, the one or more computerprograms cause the processor to perform operations to: select an imageas a reference image; extract at least one related image which isrelated to the selected reference image; and generate the new compositeimage by using the selected reference image and the extracted at leastone related image based on a layout, the layout comprising a definitionof at least one kind of information of a background image, the number ofimages to be used to generate the composite image, a size of each imageto be used to generate the composite image, and positional relationamong images to be used to generate the composite image, the imagegenerating apparatus has a playback mode where an image to be displayedby the display unit is changed to another image while an image of aplurality of images is being displayed by the display unit, and theinstructing unit issues the instruction to generate the new compositeimage at arbitrary timing while the image change is being performed inthe playback mode, the image generating apparatus has a shooting modewhere an image is shot, and the instructing unit issues the instructionto generate the new composite image in the playback mode and issues aninstruction to shoot an image in the shooting mode.
 6. The imagegenerating apparatus of claim 5 further comprising a plurality of unitswhich instruct image shooting, wherein the instructing unit is one ofthe plurality of units.
 7. The image generating apparatus of claim 5,wherein the instructing unit issues the instruction to generate the newcomposite image according to manipulation of a manipulating unitinstalled on a predetermined location of a body of the image generatingapparatus.
 8. The image generating apparatus of claim 1, wherein theprocessor performs the operation to extract the at least one relatedimage based on time-related information comprised in the referenceimage.
 9. The image generating apparatus of claim 1, wherein theprocessor performs the operation to extract the at least one relatedimage based on location information comprised in the reference image.10. The image generating apparatus of claim 1, wherein the processorfurther performs an operation to recognize a subject in the selectedreference image, and the processor performs the operation to extract theat least one related image based on the recognized subject.
 11. An imagegenerating method performed by an image generating apparatus comprising:selecting an image as a reference image from a plurality of images;selecting a most recent layout selected for a composite image generatedmost recently as a layout for a new composite image; extracting at leastone related image from the plurality of images, which is related to thereference image selected from the plurality of images, by selecting theat least one related image from the plurality of images; and generatingthe new composite image different according to the selected referenceimage, by using the selected reference image and the extracted at leastone related image based on a layout, the layout comprising at least onedefinition of information of a background image, the number of images tobe used to generate the composite image, a size of each image to be usedto generate the composite image, and positional relation among images tobe used to generate the composite image.
 12. The image generatingapparatus of claim 1, further comprising an instructing unit, wherein,according to one instruction to generate the new composite image issuedby the instructing unit, the processor performs a series of operationsto: select an image as a reference image from the plurality of images;extract the at least one related image from the plurality of images,which is related to the selected reference image, by selecting the atleast one related image from the plurality of images; and generate thenew composite image different according to the selected reference image,by using the selected reference image and the extracted at least onerelated image based on a layout, the layout comprising a definition ofat least one kind of information of a background image, the number ofimages to be used to generate the composite image, a size of each imageto be used to generate the composite image, and positional relationamong images to be used to generate the composite image, the newcomposite image being different according to the selected referenceimage.
 13. The image generating apparatus of claim 12, wherein theprocessor performs an operation to select a layout to be used whengenerating the new composite image as a part of the series of operationsaccording to the one instruction to generate the new composite image.14. The image generating apparatus of claim 1, further comprising adisplay unit for displaying an image of the plurality of images, whereinthe processor performs the operation to select an image which is beingdisplayed by the display unit as the reference image.
 15. An imagegenerating method performed by an image generating apparatus, whichcomprises a display unit and an instructing unit, comprising: accordingto an instruction to generate a new composite image issued by theinstructing unit, selecting an image as a reference image; extracting atleast one related image which is related to the selected referenceimage; and generating the new composite image by using the selectedreference image and the extracted at least one related image based on alayout, the layout comprising a definition of at least one kind ofinformation of a background image, the number of images to be used togenerate the composite image, a size of each image to be used togenerate the composite image, and positional relation among images to beused to generate the composite image, the image generating apparatus hasa playback mode where an image to be displayed by the display unit ischanged to another image while an image of a plurality of images isbeing displayed by the display unit, and the instructing unit issues theinstruction to generate the new composite image at arbitrary timingwhile the image change is being performed in the playback mode, andwhile an image is being displayed in the playback mode, the processorperforms the operation to generate the new composite image in the casethat the instructing unit issues the instruction to generate the newcomposite image, and a changed image is displayed without the generationof the new composite image being performed in the case that theinstructing unit issues an instruction to change the image beingdisplayed.